Different-view image generating apparatus and different-view image generating method

ABSTRACT

An apparatus includes: a generating unit which generates, from each of images respectively obtained at viewpoint positions, one of different-viewpoint images which corresponds to an image at a virtual viewpoint position different from the viewpoint positions, the different-viewpoint image including a hole area in which a pixel value is missing; a calculating unit which calculates, for each of processing units respectively in predetermined areas in the different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the different-viewpoint images; a calculating unit which calculates, for each processing unit, a combination ratio of the different-viewpoint image, based on the hole density of the processing unit co-located with an other one of the processing units in an other one of the different-viewpoint images; and a combining unit which combines the different-viewpoint images, based on the combination ratios.

TECHNICAL FIELD

The present disclosure relates to an image processing technique forthree-dimensional (3D) display, and particularly to adifferent-viewpoint image generating apparatus which generates, from twoor more images captured at mutually different viewpoint positions,different-viewpoint images having viewpoint positions different fromthose of the respective two or more images.

BACKGROUND ART

There are known techniques for displaying images providing a disparity(hereinafter, also referred to as stereo images) to right and left eyesof a viewer, and thereby allowing the viewer to view planar videos as astereoscopic video.

There are also techniques for generating, from a pair of stereoscopicimages, images having viewpoints different from those of the pair(different-viewpoint images) (for example, see Patent Literature 1).

The technique disclosed in Patent Literature 1 is a technique forgenerating such different-viewpoint images by shifting pixels inhorizontal directions in images according to distances in depthdirections in the images using depth maps indicating distances in thedepth directions. This technique is generally called as Depth ImageBased Rendering (DIBR).

The images generated using the DIBR may include areas which do notappear in the original stereoscopic images. The areas do not haveassigned pixel values (hereinafter, referred to as hole areas), and thusneed to be interpolated using some interpolation process.

CITATION LIST Patent Literature

[PTL 1]

Japanese Unexamined Patent Application. Publication No. 2010-218548

SUMMARY OF INVENTION Technical Problem

In view of this, the present disclosure provides a different-viewpointimage generating apparatus capable of interpolating hole areas indifferent-viewpoint images to generate high-quality images.

Solution to Problem

A different-viewpoint image generating apparatus according to an aspectof the present disclosure includes: a different-viewpoint imagegenerating unit configured to generate, from each of two or more imagesrespectively obtained at two or more viewpoint positions, one of two ormore different-viewpoint images which correspond to an image obtainableat a virtual viewpoint position different from the two or more viewpointpositions, based on distance information indicating a depth of a pixelin one of the two or more images, the one of the two or moredifferent-viewpoint images including a hole area in which a pixel valueis missing; a hole density calculating unit configured to calculate, foreach of processing units respectively in predetermined areas in the twoor more different-viewpoint images, a hole density indicating, withrespect to the predetermined area, a ratio of one of the hole areas inthe processing units in the two or more different-viewpoint images, theprocessing unit being made up of one or more pixels; a combination ratiocalculating unit configured to calculate, for each of the processingunits, a combination ratio of one of the two or more different-viewpointimages, based on the hole density of the processing unit which isco-located with an other one of the processing, units in an other one ofthe two or more different-viewpoint images; and a different-viewpointimage combining unit configured to combine the two or moredifferent-viewpoint images, based on the combination ratios of theprocessing units.

These general and specific aspects may be implemented by arbitrarilycombining a system, a method, an integrated circuit, a computer program,or a recording medium such as a CD-ROM, or any combination of systems,methods, integrated circuits, computer programs or recording media.

Advantageous Effects of Invention

The different-viewpoint image generating apparatus according to thepresent disclosure is capable of interpolating hole areas indifferent-viewpoint images to generate high-quality images.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing an overall structure of adifferent-viewpoint image generating apparatus according to anembodiment.

FIG. 2 is a diagram showing examples of images and depth maps which areinput to different-viewpoint image generating unit.

FIG. 3 is a diagram showing different-viewpoint mages which aregenerated by the different-viewpoint image generating unit.

FIG. 4 is a diagram showing a hole density calculating method performedby a hole density calculating unit.

FIG. 5 is a schematic diagram for illustrating a specific example inwhich the hole density calculating method is performed.

FIG. 6 is a diagram showing a combination ratio map which is generatedby the combination ratio calculating unit.

FIG. 7 is a diagram showing a relationship between the combination ratioα and a difference between a hole density of a left different-viewpointimage and a hole density of a right different-viewpoint image.

FIG. 8 is a schematic diagram showing a hole-embedded leftdifferent-viewpoint image and a hole-embedded right different-viewpointimage.

FIG. 9 is a diagram showing an output image which is generated by thedifferent-viewpoint image combining unit.

FIG. 10 is a flowchart of operations performed by thedifferent-viewpoint image generating apparatus according to theembodiment.

FIG. 11 is a diagram showing an overall structure of adifferent-viewpoint image generating apparatus according to an otherembodiment.

FIG. 12 is a flowchart of operations performed by thedifferent-viewpoint image generating apparatus according to the otherembodiment.

FIG. 13 is a first diagram showing an application example of any of thedifferent-viewpoint image generating apparatuses.

FIG. 14 is a second diagram showing an application example of any of thedifferent-viewpoint image generating apparatuses.

DESCRIPTION OF EMBODIMENTS (Underlying Knowledge Forming Basis of thePresent Disclosure)

There are various techniques as methods for displaying stereoscopicvideos such as 3D movies and 3D television programs. These techniquesare common in the point of allowing viewers to recognize planar videosas stereoscopic videos by displaying stereoscopic images providingdisparities to right and left eyes of viewers. A pair of stereoscopicimages having a larger disparity provides a more stereoscopic view tothe right and left eyes of a viewer when displayed. On the other hand, apair of stereoscopic images having a smaller disparity provides a lessstereoscopic view to the viewer.

A disparity between captured stereoscopic images is determined accordingto an inter-lens distance of stereo cameras used to capture stereoscopicimages. It is desirable that the disparity of the stereoscopic images bedesigned to be adjusted later to adjust stereoscopic vision.

In addition, naked-eye 3D display using lenticular lenses is fordisplaying stereoscopic images having viewpoint positions which differaccording to the positions of viewers. In such a case, in order toswitch stereoscopic images to be displayed naturally, images captured atmany viewpoint positions need to be captured in advance. However, suchimage capturing at many viewpoint positions requires high cost.Accordingly, it is necessary to generate images having differentviewpoint positions by modifying stereoscopic images.

Here, as described in the Background Art, the known DIBR is intended togenerate, from stereoscopic images, different-viewpoint images havingviewpoints different from those of the stereoscopic images.

The images generated using the DIBR may include hole areas withoutassigned pixel values, and such hole areas need to be interpolated usingsome interpolation process.

The hole areas can be simply interpolated according, to a linearinterpolation process using pixel values of pixels in peripheral areasof the hole areas. However, in the case of a large hole area, a linearinterpolation process results in a reduction in the image quality.

On the other hand, the technique disclosed in Patent Literature 1 is atechnique for interpolating a hole area in a different-viewpoint imagegenerated from a left-viewpoint image that constitutes a stereoscopicimage together with a right-viewpoint image by assigning pixel values ofa corresponding area in the right-viewpoint image.

However, since the technique disclosed in Patent Literature 1 isintended to directly assign, to a hole area in an image, the pixelvalues of pixels in an opposite-viewpoint image, the boundary betweenthe hole area and the peripheral area after the interpolation do notconnect seamlessly.

In order to solve the above problem, a different-viewpoint imagegenerating apparatus according to an aspect of the present disclosureincludes: a different-viewpoint image generating unit configured togenerate, from each of two or more images respectively obtained at twoor more viewpoint positions, one of two or more different-viewpointimages which correspond to an image obtainable at a virtual viewpointposition different from the two or more viewpoint positions, based ondistance information indicating a depth of a pixel in one of the two ormore images, the one of he two or more different-viewpoint imagesincluding a hole area in which a pixel value is missing; a hole densitycalculating unit configured to calculate, for each of processing unitsrespectively in predetermined areas in the two or moredifferent-viewpoint images, a hole density indicating, with respect tothe predetermined area, a ratio of one of the hole areas in theprocessing units in the two or more different-viewpoint images, theprocessing unit being made up of one or more pixels; a combination ratiocalculating unit configured to calculate, for each of the processingunits, a combination ratio of one of the two or more different-viewpointimages, based on the hole density of the processing unit which isco-located with an other one of the processing units in an other one ofthe two or more different-viewpoint images; and a different-viewpointimage combining unit configured to combine the two or moredifferent-viewpoint images, based on the combination ratios of theprocessing units.

In this way, it is possible to interpolate precisely the hole areasformed when the different-viewpoint images are generated. Accordingly,it is possible to generate the different-viewpoint images which havehigh quality from the two or more images.

In addition, the different-viewpoint image generating apparatus mayfurther include a hole area interpolating unit configured to interpolateeach of the hole areas in the two or more different-viewpoint images,using pixel values within a corresponding one of the two or moredifferent-viewpoint images, wherein the different-viewpoint imagecombining unit may be configured to combine the two or moredifferent-viewpoint images including the interpolated hole areas, basedon the combination ratios.

In addition, the different-viewpoint image generating apparatus mayfurther include a hole area interpolating unit configured to interpolatethe hole areas within images combined by the different-viewpoint imagecombining unit, using pixel values within the images.

In addition, the hole density calculating unit may be configured tocalculate, as a plurality of the hole densities of the co-locatedprocessing units in the two or more different-viewpoint images, aplurality of the ratios of the hole areas in windows which are thepredetermined areas having the co-located processing units as centers ofthe predetermined areas.

In addition, the hole density calculating unit may be configured tocalculate a plurality of the hole densities by adding different weightsto a hole area located at a central part of the window and a hole arealocated at a peripheral part of the window.

In addition, the combination ratio calculating unit may be configured tocalculate, for the co-located processing units in the two or moredifferent-viewpoint images, a plurality of the combination ratios whichbecome larger as a plurality of the hole densities of the co-locatedprocessing units become smaller.

In addition, when hole densities of co-located processing units areequal to each other between the two or more different-viewpoint images,the combination ratio calculating unit may be configured to calculate,for the co-located processing units having the equal hole densities,combination ratios which become larger as the positions of the two ormore different-viewpoint images become closer to the position of thevirtual viewpoint position.

In addition, the processing unit may be a pixel, the hole densitycalculating unit may be configured to calculate a hole density for eachof pixels co-located in the two or more different-viewpoint images, andthe combination ratio calculating unit may be configured to calculate,for each of the co-located pixels, the combination ratio of one of thetwo or more different-viewpoint images.

These general and specific aspects may be implemented by arbitrarilycombining a system, a method, an integrated circuit, a computer program,or a computer-readable recording medium such as a CD-ROM, or anycombination of systems, methods, integrated circuits, computer programs,or computer-readable recording media.

Hereinafter, embodiments are described with reference to the drawings.

Each of the exemplary embodiments described below shows a general orspecific example. The numerical values, shapes, materials, structuralelements, the arrangement and connection of the structural elements,steps, the processing order of the steps etc.

shown in the following exemplary embodiments are mere examples, andtherefore do not limit the present disclosure. Therefore, among thestructural elements in the following exemplary embodiments, structuralelements not recited in any one of the independent claims which definethe most generic concept are described as arbitrary structural elements.

Embodiment

FIG. 1 is a diagram showing an overall structure of adifferent-viewpoint image generating apparatus 100 according to anembodiment.

The different-viewpoint image generating apparatus 100 includes a leftdifferent-viewpoint image generating unit 101, a rightdifferent-viewpoint image generating unit 102, a hole densitycalculating unit 103, a combination ratio calculating unit 104, a holearea interpolating unit 105, and a different-viewpoint image combiningunit 106. This embodiment describes an example in which thedifferent-viewpoint image generating apparatus 100 generatesdifferent-viewpoint images from a left-viewpoint image and aright-viewpoint image which constitute a stereoscopic image.

The left different-viewpoint image generating unit 101 generates a leftdifferent-viewpoint image by shifting each of pixels in theleft-viewpoint image in a horizontal direction based on the depth of thepixel, based on the left-viewpoint image and a depth map (a left depthmap) of the left-viewpoint image.

The right different-viewpoint image generating unit 102 generates aright different-viewpoint image by shifting each of pixels in theright-viewpoint image in a horizontal direction based on the depth ofthe pixel, based on the right-viewpoint image and a depth map (a rightdepth map) of the right-viewpoint image.

Here the left-viewpoint image is an image captured at a left-viewpointposition, and the right-viewpoint image is an image captured at aright-viewpoint position different from the left-viewpoint position. Inaddition, the left different-viewpoint image and the rightdifferent-viewpoint image are images corresponding to an imageobtainable at the same virtual viewpoint position (the positiondifferent from any of the left-viewpoint position and theright-viewpoint position). The depth map is distance informationindicating the depth of each pixel in each of the images (the distancefrom a viewpoint position to a subject presented by the pixel).

The left different-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generating unit 102 do not always need to beconfigured separately as shown in FIG, 1. In other words, a singledifferent-viewpoint image generating unit may generate a leftdifferent-viewpoint image and a right different-viewpoint image from theleft- and right-viewpoint images and the depth maps thereof. Morespecifically, the different-viewpoint image generating unit generates,from each of the two images obtained at two viewpoint positions, onedifferent-viewpoint image corresponding to an image obtainable at avirtual viewpoint position different from the viewpoint position of thecorresponding one of the two images, based on distance informationindicating the depth of each pixel in the image.

The hole density calculating unit 103 generates a hole density mapindicating a distribution of hole areas for each of processing units ineach of the left different-viewpoint image and the rightdifferent-viewpoint image respectively generated by the leftdifferent-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generating unit 102. More specifically, thehole density calculating unit 103 calculates, for each of the generatedtwo different-viewpoint images, a hole density which is an occupationpercentage of the hole areas in a predetermined area including theprocessing unit, for each processing unit made up of one or more pixels.Here, a hole area is an area without any pixel value in adifferent-viewpoint image.

In this embodiment, the processing unit is made of one or more pixels,The hole density calculating unit 103 calculates a hole density for eachpixel in each of the two different-viewpoint images. In this embodiment,a later-described window is used as the predetermined area.

The combination ratio calculating unit 104 generates a combinationratio, map indicating a ratio at the time of combining the leftdifferent-viewpoint image and the right different-viewpoint image, basedon the hole density maps generated by the hole density calculating unit103. More specifically, the combination ratio calculating unit 104compares, for each set of co-located pixels (processing units) in thetwo different-viewpoint images, the magnitudes of the hole densities ofthe pixels in the pair, and calculates, for each pair of the pixels(processing unit), a combination ratio between the twodifferent-viewpoint images according to the magnitudes of the comparedhole densities.

The hole area, interpolating unit 105 performs a hole embedment processfor interpolating a hole area in an image using pixel values locatedaround the hole area within the same image, for each of the leftdifferent-viewpoint image and the right different-viewpoint imagerespectively generated by the left different-viewpoint image generatingunit 101 and the right different-viewpoint image generating unit 102. Inother words, the hole area interpolating unit 105 interpolates each ofthe hole areas of the two different-viewpoint images using the pixelvalues within the corresponding one of the different-viewpoint images.

Here, a pixel value is information indicating at least a luminance and acolor of a pixel. Specifically, the pixel value is information composedof luminance values of RGB color components, information composed ofluminance and chrominance values, or the like. In addition, the pixelvalues may include supplemental information such as depth values inaddition to information related to colors.

The different-viewpoint image combining unit 106 generates an outputimage (an output different-viewpoint image) by combining the leftdifferent-viewpoint image and the right different-viewpoint image afterthe hole embedment process by the hole area interpolating unit 105,based on the combination ratio shown by the combination ratio mapgenerated by the combination ratio calculating unit 104, Morespecifically, the different-viewpoint image combining unit 106 combinesthe two different-viewpoint images with the hole areas interpolated,based on the calculated combination ratio.

Here, the structural elements of the different-viewpoint imagegenerating apparatus 100 may be configured with an exclusive hardware,or may be realized by executing a software program suitable for each ofthe structural elements. Each of the structural elements may be realizedby means of the program executing unit such as a CPU or a processorreading and executing a software program recorded on a recording mediumsuch as a hard disk or a semiconductor memory.

Next, a detailed description is given of operations performed by thestructural elements of the different-viewpoint image generatingapparatus 100, First, detailed descriptions are given of how the leftdifferent-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generating unit 102 perform processes forgenerating different-viewpoint images.

FIG. 2 is a diagram showing examples of images and depth maps to beinput to the left different-viewpoint image generating unit 101 and theright different-viewpoint image generating unit 102.

The left-viewpoint image 211 is an image obtained by capturing a subject201 and a subject 202 at a left-viewpoint position 210. Theright-viewpoint image 221 is an image obtained by capturing a subject201 and a subject 202 at a right-viewpoint position 220. Theleft-viewpoint image 211 and the right-viewpoint image 221 are imagesfor stereoscopic viewpoints obtained by capturing images of the samesubjects 201 and 202 located at relatively different positions in theimages.

The left depth map 212 is an image showing the background of aleft-viewpoint image 211 and the depth of the subject 201. The rightdepth map 222 is an image showing the background of a right-viewpointimage 221 and the depth of the subject 201. In other words, the leftdepth map 212 shows a distance from the left-viewpoint position 210 tothe subject 201, and the right depth map 222 shows a distance from theright-viewpoint position 220 to the subject 201. For simplification, thedepth of the subject 202 is not shown in this embodiment although it isalso shown in the left depth map 212 and the right depth map 222 inreality.

Here, in the depth map in this embodiment, a pixel presenting a subjectlocated closer to a viewpoint position is a bright pixel having a higherluminance value, and a pixel presenting a subject located farther fromthe viewpoint position is a dark pixel having a lower luminance value.

The left different-viewpoint image generating unit 101 generates a leftdifferent-viewpoint image corresponding to an image obtainable at avirtual viewpoint position, from the left-viewpoint image and the leftdepth map as shown in FIG. 2. Likewise, the right different-viewpointimage generating unit 102 generates a right different-viewpoint imagecorresponding to an image obtainable at a virtual viewpoint position,from the right-viewpoint image and the right depth map as shown in FIG.2.

FIG. 3 is a diagram showing the left different-viewpoint image generatedby the left different-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generated by the right different-viewpointimage generating unit 102.

In general, the subject 201 closer from the viewpoint position has alarge amount of shift due to the difference in the viewpoint positionthan the subject 202 farther from the viewpoint position. Inconsideration of this, the left different-viewpoint image generatingunit 101 and the right different-viewpoint image generating unit 102shift the pixels in the input images in the horizontal directionconsidering the distances from the viewpoint position of the inputimages (the left-viewpoint image 211 and the right-viewpoint image 221)to the virtual viewpoint position. At this time, the leftdifferent-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generating unit 102 adjust the amount of shiftof each pixel according to the distance in the depth direction in theimage using the depth map showing the distance in the depth direction.

As shown in FIG. 3, the left different-viewpoint image generating unit101 generates a left different-viewpoint image 302 corresponding to animage obtainable by capturing the images of the subjects 201 and 202 atthe virtual viewpoint position 301. In addition, the rightdifferent-viewpoint image generating unit 102 generates a rightdifferent-viewpoint image 303 obtainable by capturing the images of thesubjects 201 and 202 at the virtual viewpoint position 301.

Here, as described above, when the viewpoint position is shifted fromthe left-viewpoint position 210 to the virtual-viewpoint position 301,the amount of shift of the subject 201 located frontward is larger thanthe amount of shift of the subject 202 located backward. The sameapplies to a case where a viewpoint position is shifted from theright-viewpoint position 220 to the virtual viewpoint position 301,except for the point that the shift direction is opposite. In otherwords, the shift amount Δ×of the pixel value of each pixel with theshift of the viewpoint position is calculated according to Expression 1below.

Δx=d·Δb   (Expression 1)

Here, d denotes a depth value (a value in the depth map) of each pixel,and decreases for a backward viewpoint position and increases for afrontward viewpoint position. Here, Δ b denotes the amount of shift inthe viewpoint position (either the amount of shift from theleft-viewpoint position 210 to the virtual viewpoint position 301 or theamount of shift from the right-viewpoint position 220 to the virtualviewpoint position 301).

A pixel value having a larger depth value d has a larger value of theamount of shift Δ x, resulting in a hole area without any pixel value ata side of the subject 201 in each of the left different-viewpoint image302 and the right different-viewpoint image 303 after the shift ofviewpoints. More specifically, the left different-viewpoint image 302generated from the left-viewpoint image 211 has a hole area 310 at theright side of the subject 201 in the image, and the rightdifferent-viewpoint image 303 generated from the right-viewpoint image221 has a hole area 311 at the left side of the subject 201 in theimage.

In reality, a hole area is generated also at a side of the subject 202in the image except for a case where the subject 202 is located at thesame distance as the background. However, in FIG. 3, the hole areagenerated around the subject 202 is not shown.

Next, a description is given of how the hole density calculating unit103 performs a hole density calculating method. FIG. 4 is a diagramshowing the hole density calculating method performed by the holedensity calculating unit 103.

The hole density calculating unit 103 scans a window 403 in each of ahole map 401 ((a) in FIG. 4) obtained by extracting only the hole areasfrom the left different-viewpoint image 302 and a hole map 402 ((b) inFIG. 4) obtained by extracting only the hole areas from the rightdifferent-viewpoint image 303. Next, the hole density calculating unit103 calculates, as a hole density Den, an occupation percentage of thehole area with respect to the whole window 403 when the pixel isoverlapped with the center of the window 403, for each pixel (processingunit) in relation to each of the hole maps 401 and 402. The hole densitycalculating unit 103 calculates the hole density maps 406 and 407composed of the calculated hole densities, according to Expression 2below.

Den(x, y)=ΣisHole(H[x+dx, y+dy])/N   (Expression 2)

In Expression 2, N denotes a total number of pixels in the window,H[x+dx, y+dy] denotes components on the coordinates [x+dx, y+dy] in thehole map. In addition, dx and dy denote positions relative to the centerof the window.

The hole density calculating unit 103 calculates the hole density Den ofthe pixel located at the coordinates (x, y) by calculating a totalnumber of hole areas presented by H[x+dx, y+dy] and dividing the totalnumber by N. For example, in FIG. 4, the hole density of the window 404in the hole map 401 is smaller than the hole density of the window 405in the hole map 402.

Hereinafter, the hole density calculating method is described using aspecific example as shown in FIG, 5. FIG. 5 is a schematic diagram forillustrating the specific example in which the hole density calculatingmethod is performed.

In (a) of FIG. 5, a hole area 409 of 4×5 pixels is included in a holemap 408. The hole density of a pixel A is calculated as the number ofpixels included in the hole area 409 with respect to a window 410 a of5×5 pixels having the pixel A as its center. In other words, the holedensity of the pixel A is calculated as 3/25=0.12. Likewise, the holedensity of a pixel B is calculated as the number of pixels included inthe hole area 409 with respect to a window 410 b and calculated as10/25=0.4.

Here, the window may have a square or rectangular shape. The window mayhave a round or oval shape, without being limited to the rectangularshape.

In addition, the hole density calculating unit 103 may calculate a holedensity Den by assigning different weights to a hole area located at acentral part of a window and a hole area located at a peripheral part ofthe window.

For example, as shown in the calculation of the hole density of a pixelB in (b) of FIG. 5, the hole density calculating unit 103 calculates apixel as two points for the hole area located at the central part of thewindow 401 b, and calculates a pixel as one point for the hole arealocated at the peripheral part of the window 410 b. In this way, alater-described combination ratio is calculated considering thepositions of the hole areas within the window.

In addition, the size of the window may be specified by a user byinputting a parameter to the different-viewpoint image generatingapparatus 100 of the user.

As described above, the hole density calculating unit 103 calculateshole densities for each of pixels in the left different-viewpoint image302 and the right different-viewpoint image 303. The hole density map406 ((c) of FIG. 4) is an image presenting the hole density calculatedfor each pixel in the left different-viewpoint image 302, and the holedensity map 407 ((d) of FIG. 4) is an image presenting the hole densitycalculated for each pixel in the right different-viewpoint image 303.

As shown in the hole density maps 406 and 407, the pixel located aroundthe edge of the hole area has a low hole density, and the pixel locatedaround the center of the hole area has a high hole density.

Next, a description is given of the combination ratio calculating methodperformed by the combination ratio calculating unit 104. The combinationratio calculating unit 104 compares, for each pair of co-located pixels,hole densities in the hole density maps 406 and 407, and calculates thecombination ratios (weights for mixture of the pixel values) to be usedto combine the left-viewpoint image and the right-viewpoint image. As aresult, a combination ratio map 510 as shown in FIG. 6 is generated.

In this embodiment, the combination ratio α takes a value from 0 to 1,and indicates a combination ratio of the left different-viewpoint image302 with respect to the right different-viewpoint image 303. In otherwords, the pixel value of the left different-viewpoint image 302 isdirectly used for a pixel of α=1 instead of a pixel value of the rightdifferent-viewpoint image 303. On the other hand, the pixel value of theright different-viewpoint image 303 is directly used for a pixel of α=0instead of a pixel value of the left different-viewpoint image 302.

The combination ratio α is calculated according to Expression 3 below

α=1:0≧Den(L)−Den(R)

α=0:Den(L)−Den(R)≧T

α=1−((Den(L)−Den(R))/T):

0<Den(L)−Den(R)<T   (Expression 3)

In Expression 3, Den (L) denotes the hole density of the leftdifferent-viewpoint image 302, Den (R) denotes the hole density of theright different-viewpoint image 303, and T denotes a threshold value. Inaddition, FIG. 7 is a diagram showing a difference between the holedensity of the left different-viewpoint image 302 and the hole densityof the right different-viewpoint image 303 calculated according toExpression 3, and a combination ratio α.

As for a pixel of Den(L)−Den(R)≦0, the combination ratio calculatingunit 104 determines the combination ratio α of the left-viewpoint imageto be 1. As for a pixel satisfying Den(L)−Den(R)≧a threshold value T,the combination ratio calculating unit 104 determines the combinationratio α of the left-viewpoint image to be 0. As for a pixel satisfying0<Den(L)−Den(R)<T, the combination ratio calculating unit 104 determinesthe combination ratio linearly according to the magnitudes of Den (L)and Den (R).

In this way, the combination ratio calculating unit 104 calculates, forthe set of co-located pixels, combination ratios of the twodifferent-viewpoint images such that a larger one of the combinationratios is assigned to one of the different-viewpoint images which has acomparatively larger one of the hole densities.

Here, in the combination map 510 shown in FIG. 6, an area 511 (anon-white area) is an area of α<1, and the other area (a white area) ispresented as α=1.

Here, in Expression 3, the threshold value T may be changed by the userby inputting a parameter to the different-viewpoint image generatingapparatus 100 of the user. In addition, for a pixel satisfying0<Den(L)−Den(R)<T, the combination ratio α may be determined linearly.

In this embodiment, as shown in FIG. 3, the distance from theleft-viewpoint position 210 to the virtual-viewpoint position 301 iscloser to the distance from the right-viewpoint position 220 to thevirtual-viewpoint position 301. Thus, in principle, the pixel value ofthe left different-viewpoint image 302 having fewer hole areas is usedas the pixel value of the different-viewpoint images after thecombination.

Thus, the graph 500 shown in FIG. 7 has the horizontal axis presentingDen (L)−Den (R) (which is an expression as a basis of Expression 3). Inthe opposite case, when the distance from the left-viewpoint position210 to the virtual-viewpoint position 301 is larger than the distancefrom the right-viewpoint position 220 to the virtual-viewpoint position301, the graph as shown in FIG. 7 has the horizontal axis presenting Den(R)−Den (L) (which is an expression as a basis of Expression 3).

In this way, when Den(L)=Den(R) is satisfied, the combination ratio forthe pixel value of the left different-viewpoint image 302 is large. Inother words, when the hole densities are equal to each other, thecombination ratio calculating unit 104 calculates, for each of the pairof pixels, a combination ratio of the two different-viewpoint imagessuch that a larger one of the combination ratios is assigned to one ofthe different-viewpoint images which has a viewpoint position closer tothe virtual-viewpoint position 301.

In this way, as a result, a larger number of pixel values in the imageincluding a smaller number of hole areas are used in thedifferent-viewpoint image (output image) after the combination. Thus,the different-viewpoint image generating apparatus 100 can generate theoutput image with a higher precision.

Here, the combination ratio calculating method may be any othercalculating method, without being limited to the one according toExpression 3.

Next, a description is given of a hole area interpolation methodperformed by the hole area interpolating unit 105. The hole areainterpolating unit 105 obtains the left different-viewpoint image 302and the right different-viewpoint image 303 respectively from the leftdifferent-viewpoint image generating unit 101 and the rightdifferent-viewpoint image generating unit 102, and performs the holeembedment process (interpolation process) on the hole areas of the leftdifferent-viewpoint image 302 and the right different-viewpoint image303.

In this embodiment, the hole area interpolating unit 105 performs a holeembedment process using pixel values of pixels around a hole area. FIG.8 is a schematic diagram showing a hole-embedded leftdifferent-viewpoint image (hereinafter also referred to as a leftinterpolated image) and a hole-embedded right different-viewpoint image(hereinafter also referred to as a right interpolated image).

FIG. 8 shows an example where the left interpolated image 600 ((a) inFIG. 8) and a right interpolated image 610 ((b)) in FIG. 8) in the casewhere the hole area interpolating unit 105 performs a horizontalinterpolation process (a linear interpolation process in a horizontaldirection) using pixel values of pixels adjacent in the horizontaldirection. When the horizontal interpolation process is used, aninterpolated area 601 (which was a hole area originally) of the leftinterpolated image 600 shown in (a) of FIG. 8 is interpolated such thatthe pixel values of the pixels located left and right of the hole areaare extended. This is true of an interpolated area 611 of the rightinterpolated image 610 shown in (b) of FIG. 8.

Here, the interpolation process (intra interpolation process) of thehole area interpolating unit 105 may be a process other than the linearinterpolation process. For example, it is also good to compare depthvalues in depth maps at sets of coordinates adjacent to the hole area inthe horizontal direction, and perform an extrapolation process usingpixel values of pixels farther from the viewpoint.

Next, a description is given of an image combining method performed bythe different-viewpoint image combining unit 106. Thedifferent-viewpoint image combining unit 106 obtains the leftinterpolated image 600 and the right interpolated image 610, and obtainsthe combination ratio map 510 from the hole density calculating unit103. Next, the different-viewpoint image combining unit 106 generates anoutput image obtained by combining the two interpolated images (the leftinterpolated image 600 and the right interpolated image 610). FIG. 9 isa diagram showing an output image generated by the different-viewpointimage combining unit 106.

When the left interpolated image 600, the right interpolated image 610,and the combination ratio map 510 are respectively denoted as L (x, y),R (x, y), and α (x, y), an output image 700 (O (x, y)) is calculatedaccording to Expression 4 below.

O(x, y)=L(x, y)·α(x, y)+R(x, y)·{1−α(x, y)}  (Expression 4)

The right side area 701 of a subject 201 in the output image 700 shownin FIG. 9 corresponds to an area 511 in the combination ratio map 510.

The pixels located at the central part of the right side area 701 areassigned with pixel values of the right interpolated image 610. Thepixels located at the remaining area other than the right side area 701are assigned with pixel values of the left interpolated image 600. Thepixels located at the peripheral part of the right side area 701 areassigned with pixel values obtained by mixing the pixel values of theleft interpolated image 600 and the pixel values of the rightinterpolated image 610 according to the combination ratio α.

Lastly, an order of operations by the different-viewpoint imagegenerating apparatus 100 is described with reference to FIG. 10. FIG. 10is a flowchart of operations by the different-viewpoint image generatingapparatus 100.

First, the left different-viewpoint image generating unit 101 and theright different-viewpoint image generating unit 102 generate the leftdifferent-viewpoint image 302 and the right different-viewpoint image303 from the input images (a left-viewpoint image 211 and aright-viewpoint image 221) and the depth maps thereof (S101).

Next, the hole density calculating unit 103 calculates, for each pixel,a hole density in each hole area in each of the left different-viewpointimage 302 and the right different-viewpoint image 303 (S102). Next, thecombination ratio calculating unit 104 calculates the combination ratiosat the time of combining the left different-viewpoint image 302 and theright different-viewpoint image 303 (S103).

In addition, the hole area interpolating unit 105 interpolates the holeareas in the left different-viewpoint image 302 and the rightdifferent-viewpoint image 303 (S104).

Lastly, the different-viewpoint image combining unit 106 combines theleft different-viewpoint image and right different-viewpoint imageinterpolated in Step S104, based on the combination ratios calculated inStep S103 (S105).

Here, the order of Step S102, Step S103, and Step S104 is notspecifically limited. The different-viewpoint image generating apparatus100 may perform Step S102, Step S103, and Step S104 in this order, ormay perform Step S104, Step S102, and Step S103 in this order.Alternatively, the processes in Step S102, Step S103, and Step S104 maybe performed in parallel.

As described above, the different-viewpoint image generating apparatus100 calculates the combination ratio map 510, using the hole densitycalculating unit 103 and the combination ratio calculating unit 104.Next, the different-viewpoint image combining unit 106 combines the twodifferent-viewpoint images generated from the left-viewpoint image andthe right-viewpoint image according to the calculated combinationratios.

In this way, an interpolation process is performed only on an area whichrequires interpolation in a hole area, and the combination ratioscalculated based on a hole density is reflected in the interpolationprocess. Thus, the hole area is interpolated smoothly. In short, withthe different-viewpoint image generating apparatus 100, it is possibleto generate a high-quality image by interpolating the hole area in thedifferent-viewpoint image. Stated differently, the different-viewpointimage generating apparatus 100 is capable of generating a high-qualitydifferent-viewpoint image (an output image).

Other Embodiments

The above embodiment has been described as an example of a techniquedisclosed in the present application. The technique in the presentdisclosure is not limited thereto, and is applicable to embodimentsobtainable by arbitrarily providing some modification, replacement,addition, omission, etc, to the exemplary embodiment. In addition, it isalso possible to conceive new embodiments by arbitrarily combining thestructural elements described in the above embodiment.

In the above embodiment, two different-viewpoint images withinterpolated hole areas are combined. However, it is also good tocombine two different-viewpoint images with hole areas first, and theninterpolate the hole areas in the combined image.

FIG. 11 is a diagram showing an overall structure of adifferent-viewpoint image generating apparatus 100 a according toanother embodiment. FIG. 12 is a flowchart of operations performed bythe different-viewpoint image generating apparatus 100 a. It is to benoted that differences from the different-viewpoint image generatingapparatus 100 are mainly described below, and the same descriptions asin the above embodiment are not repeated below.

A different-viewpoint image combining unit 106 combines a leftdifferent-viewpoint image 302 and a right different-viewpoint image 303according to combination ratios calculated by a combination ratiocalculating unit 104 (S106 in FIG. 12). At this time, when both ofcorresponding pixels (co-located pixels having the same coordinates) ofthe left different-viewpoint image 302 and the right different-viewpointimage 303 are hole areas, the corresponding pixels are handled as holeareas. Accordingly, the image combined by the different-viewpoint imagecombining unit 106 includes the hole areas.

Next, the hole area interpolating unit 105 a interpolates the hole areasin the image combined by the different-viewpoint image combining unit106 using pixel values in the image (S107 in FIG. 12). The interpolationmethod for use at this time may be any known approach as described inthe above embodiment.

The different-viewpoint image generating apparatus 100 a is configuredto be able to combine two different-viewpoint images by prioritizingpixel values in one of the images with a smaller number of hole areas.Accordingly, the different-viewpoint image generating apparatus 100 a iscapable of generating a high-quality different-viewpoint image (anoutput image).

In addition, in the above embodiment, each of the different-viewpointimage generating apparatuses 100 and 100 a generates, from two imagescaptured at mutually different viewpoint positions, twodifferent-viewpoint images having viewpoint positions different fromthose of the two images. However, each of the different-viewpoint imagegenerating apparatuses 100 and 100 a may generate, from three or moreimages captured at mutually different viewpoint positions, three or moredifferent-viewpoint images having viewpoint positions different fromthose of the two images.

More specifically, for example, each of the different-viewpoint imagegenerating apparatuses 100 and 100 a may generate, from three imagescaptured at mutually different viewpoint positions, threedifferent-viewpoint images having viewpoint positions different fromthose of the three images. In this case, the combination ratiocalculating unit 104 calculates the combination ratios of thedifferent-viewpoint images in proportion to the values of hole densitiesof the three images.

In the above embodiment, the hole density is calculated for each pixel.However, a hole density may be calculated for each processing unitcomposed of one or more pixels (for example, the processing unit is a4×4 pixel block). In this case, a window is set for each processingunit, and hole densities are compared for each processing unit.

Each of the different-viewpoint image generating apparatuses 100 and 100a generates different-viewpoint images between two viewpoint positionsfrom two images captured at mutually different viewpoint positions inthe above embodiment, but it is capable of generatingdifferent-viewpoint images between viewpoint positions other than thetwo viewpoint positions. In addition, the two images do not always needto be images captured for stereoscopic viewpoint, and may be imagesincluding an identical subject.

For example, each of the different-viewpoint image generatingapparatuses 100 and 100 a is realized as a television receiver 800 asshown in FIG. 13. In this case, each of the different-viewpoint imagegenerating apparatuses 100 and 100 a generates different-viewpointimages from the two images captured in advance for stereoscopicviewpoint. Next, each of the different-viewpoint image generatingapparatuses 100 and 100 a is capable of displaying, for a user, acombination of two images selected from among the generateddifferent-viewpoint images and the two images captured in advance forstereoscopic view. In this way, it is possible to configure thetelevision receiver 800 which allows a user to adjust the disparitybetween images to be displayed, using a remote controller.

Alternatively, each of the different-viewpoint image generatingapparatuses 100 and 100 a may be realized as, for example, a Blu-Ray(registered trademark) player 810. In this case, each of thedifferent-viewpoint image generating apparatuses 100 and 100 a generatesdifferent-viewpoint mages from two images for stereoscopic view recordedon the Blu-Ray (registered trademark) disc to be mounted therein.Alternatively, each of the different-viewpoint image generatingapparatuses 100 and 100 a may be realized as a set top box 820. In thiscase, each of the different-viewpoint image generating apparatuses 100and 100 a generates different-viewpoint images from two images forstereoscopic view obtained through cable broadcasting or the like.

Alternatively, each of the different-viewpoint image generatingapparatuses 100 and 100 a may be realized as a digital still camera(DSC) having a 3D image capturing function as shown in (a) in FIG. 14,or a digital video camera having a 3D image capturing function as shownin (b) in FIG, 14. Each of the different-viewpoint image generatingapparatuses 100 and 100 a generates different-viewpoint images from thetwo images captured in advance for stereoscopic view.

In addition, each of the different-viewpoint image generatingapparatuses 100 and 100 a may be realized using a sever and clientsystem.

The different-viewpoint images generated by the different-viewpointimage generating apparatuses 100 and 100 a are mainly used for disparityadjustment as described above, but may be used for any other purposes.

In addition, the present disclosure includes the cases described below.

(1) Each of the different-viewpoint image generating apparatuses is,specifically, a computer system including a microprocessor, a ROM, aRAM, a hard disk unit, a display unit, a keyboard, a mouse, and so on. Acomputer program is stored in the RAM or hard disk unit. Thedifferent-viewpoint image generating apparatus achieves its functionsthrough the microprocessor's operations according to the computerprogram. Here, in order to achieve predetermined functions, the computerprogram is configured by combining plural instruction codes indicatinginstructions for the computer.

(2) A part or all of the structural elements of the different-viewpointimage generating apparatus may be configured with a single system-LSI(Large-Scale Integration). The system-LSI is a super-mufti-function LSImanufactured by integrating structural units on a single chip, and isspecifically a computer system configured to include a microprocessor, aROM, a RAM, and so on. A computer program is stored in the RAM. Thesystem-LSI achieves its function through the microprocessor's operationsaccording to the computer program.

(3) A part or all of the constituent elements constituting thedifferent-viewpoint image generating apparatus may be configured as anIC card which can be attached to and detached from the respectiveapparatuses or as a stand-alone module. The IC card or the module is acomputer system configured from a microprocessor, a ROM, a RAM, and soon, The IC card or the module may also be included in the aforementionedsuper-multi-function LSI. The IC card or the module achieves itsfunctions through the microprocessor's operations according to thecomputer program. The IC card or the module may also be implemented tobe tamper-resistant.

(4) The present disclosure may be any of the methods described above. Inaddition, any of the methods may be implemented as computer programs forexecuting the above-described method, using a computer, and may also beimplemented as digital signals including the computer programs.

Furthermore, the present disclosure may also be implemented as computerprograms or digital signals recorded on computer-readable recordingmedia such as a flexible disc, a hard disk, a CD-ROM, an MO, a DVD, aDVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), and asemiconductor memory. Furthermore, the present disclosure may also beimplemented as the digital signals recorded on these recording media.

Furthermore, the present disclosure may also be implemented as theaforementioned computer programs or digital signals transmitted via atelecommunication line, a wireless or wired communication line, anetwork represented by the Internet, a data broadcast, and so on.

The present disclosure may also be implemented as a computer systemincluding a microprocessor and a memory, in which the memory stores theaforementioned computer program and the microprocessor operatesaccording to the computer program.

Furthermore, it is also possible to execute another independent computersystem by transmitting the programs or the digital signals recorded onthe aforementioned recording media, or by transmitting the programs ordigital signals via the aforementioned network and the like.

(5) The embodiments and variations thereof may be arbitrarily combined.

It is to be noted that the present disclosure is not limited to theembodiments and variations thereof. The present disclosure includesvarious kinds of modifications that would be conceived by any personskilled in the art and made to the embodiments and variations thereofand other embodiments that would be configured by any person skilled inthe art by combining the structural elements in different embodimentsand variations thereof, without deviating from the scope of the presentinvention.

INDUSTRIAL APPLICABILITY

With the different-viewpoint image generating apparatus and thedifferent-viewpoint image generating method according to the presentdisclosure, it is possible to generate high-quality different-viewpointimages from images including depth map information captured by animaging device. The configurations thereof are applicable, for example,to consumer or industrial imaging devices (digital still cameras andvideo cameras), or devices such as mobile terminals.

REFERENCE SIGNS LIST

-   100, 100 a Different-viewpoint image generating apparatus-   101 Left different-viewpoint image generating unit-   102 Right different-viewpoint image generating unit-   103 Hole density calculating unit-   104 Combination ratio calculating unit-   105, 105 a Hole area interpolating unit-   106 Different-viewpoint image combining unit-   201, 202 Subject-   210 Left-viewpoint position-   211 Left-viewpoint image-   212 Left depth map-   220 Right-viewpoint position-   221 Right-viewpoint image-   222 Right depth map-   301 Virtual-viewpoint position-   302 Left different-viewpoint image-   303 Right different-viewpoint image-   310, 311, 409 Hole area-   401, 402, 408 Hole map-   403, 404, 405, 410 a, 410 b window-   406, 407 Hole density map-   500 Graph-   510 Combination ratio map-   511 Area-   600 Left interpolated image-   610 Right interpolated image-   601, 611 Interpolation target area-   700 Output image-   701 Right side area-   800 Television receiver-   810 Blu-Ray (registered trademark) player-   820 Set top box

1. A different-viewpoint image generating apparatus comprising: adifferent-viewpoint image generating unit configured to generate, fromeach of two or more images respectively obtained at two or moreviewpoint positions, one of two or more different-viewpoint images whichcorrespond to an image obtainable at a virtual viewpoint positiondifferent from the two or more viewpoint positions, based on distanceinformation indicating a depth of a pixel in one of the two or moreimages, the one of the two or more different-viewpoint images includinga hole area in which a pixel value is missing; a hole densitycalculating unit configured to calculate, for each of processing unitsrespectively in predetermined areas in the two or moredifferent-viewpoint images, a hole density indicating, with respect tothe predetermined area, a ratio of one of the hole areas in theprocessing units in the two or more different-viewpoint images, theprocessing unit being made up of one or more pixels; a combination ratiocalculating unit configured to calculate, for each of the processingunits, a combination ratio of one of the two or more different-viewpointimages, based on the hole density of the processing unit which isco-located with an other one of the processing units in an other one ofthe two or more different-viewpoint images; and a different-viewpointimage combining unit configured to combine the two or moredifferent-viewpoint images, based on the combination ratios of theprocessing units.
 2. The different-viewpoint image generating apparatusaccording to claim 1, further comprising a hole area interpolating unitconfigured to interpolate each of the hole areas in the two or moredifferent-viewpoint images, using pixel values within a correspondingone of the two or more different-viewpoint images, wherein thedifferent-viewpoint image combining unit is configured to combine thetwo or more different-viewpoint images including the interpolated holeareas, based on the combination ratios.
 3. The different-viewpoint imagegenerating apparatus according to claim 1, further comprising a holearea interpolating unit configured to interpolate the hole areas withinimages combined by the different-viewpoint image combining unit, usingpixel values within the images.
 4. The different-viewpoint imagegenerating apparatus according to claim 1, wherein the hole densitycalculating unit is configured to calculate, as a plurality of the holedensities of the co-located processing units in the two or moredifferent-viewpoint images, a plurality of the ratios of the hole areasin windows which are the predetermined areas having the co-locatedprocessing units as centers of the predetermined areas.
 5. Thedifferent-viewpoint image generating apparatus according to claim 4,wherein the hole density calculating unit is configured to calculate aplurality of the hole densities by adding different weights to a holearea located at a central part of the window and a hole area located ata peripheral part of the window.
 6. The different-viewpoint imagegenerating apparatus according to claim 1, wherein the combination ratiocalculating unit is configured to calculate, for the co-locatedprocessing units in the two or more different-viewpoint images, aplurality of the combination ratios which become larger as a pluralityof the hole densities of the co-located processing units become smaller.7. The different-viewpoint image generating apparatus according to claim1, wherein, when hole densities of co-located processing units are equalto each other between the two or more different-viewpoint images, thecombination ratio calculating unit is configured to calculate, for theco-located processing units having the equal hole densities, combinationratios which become larger as the positions of the two or moredifferent-viewpoint images become closer to the position of the virtualviewpoint position.
 8. The different-viewpoint image generatingapparatus according to claim 1, wherein the processing unit is a pixel,the hole density calculating unit is configured to calculate a holedensity for each of pixels co-located in the two or moredifferent-viewpoint images, and the combination ratio calculating unitis configured to calculate, for each of the co-located pixels, thecombination ratio of one of the two or more different-viewpoint images.9. A different-viewpoint image generating method comprising: generating,from each of two or more images respectively obtained at two or moreviewpoint positions, one of two or more different-viewpoint images whichcorrespond to an image obtainable at a virtual viewpoint positiondifferent from the two or more viewpoint positions, based on distanceinformation indicating a depth of a pixel in one of the two or moreimages, the one of the two or more different-viewpoint images includinga hole area in which a pixel value is missing; calculating, for each ofprocessing units respectively in predetermined areas in the two or moredifferent-viewpoint images, a hole density indicating, with respect tothe predetermined area, a ratio of one of the hole areas in theprocessing units in the two or more different-viewpoint images, theprocessing unit being made up of one or more pixels; calculating, foreach of the processing units, a combination ratio of one of the two ormore different-viewpoint images, based on the hole density of theprocessing unit which is co-located with an other one of the processingunits in an other one of the two or more different-viewpoint images; andcombining the two or more different-viewpoint images, based on thecombination ratios of the processing units.
 10. A non-transitorycomputer-readable recording medium for use in a computer, the recordingmedium having a computer program recorded thereon for causing thecomputer to execute the different-viewpoint image generating methodaccording to claim
 9. 11. An integrated circuit comprising: adifferent-viewpoint image generating unit configured to generate, fromeach of two or more images respectively obtained at two or moreviewpoint positions, one of two or more different-viewpoint images whichcorrespond to an image obtainable at a virtual viewpoint positiondifferent from the two or more viewpoint positions, based on distanceinformation indicating a depth of a pixel in one of the two or moreimages, the one of the two or more different-viewpoint images includinga hole area in which a pixel value is missing; a hole densitycalculating unit configured to calculate, for each of processing unitsrespectively in predetermined areas in the two or moredifferent-viewpoint images, a hole density indicating, with respect tothe predetermined area, a ratio of one of the hole areas in theprocessing units in the two or more different-viewpoint images, theprocessing unit being made up of one or more pixels; a combination ratiocalculating unit configured to calculate, for each of the processingunits, a combination ratio of one of the two or more different-viewpointimages, based on the hole density of the processing unit which isco-located with an other one of the processing units in an other one ofthe two or more different-viewpoint images; and a different-viewpointimage combining unit configured to combine the two or moredifferent-viewpoint images, based on the combination ratios of theprocessing units.